Tag: LLM inference
8 reviews
Synthetic workload generation for benchmarking LLM inference
Which open-source tools and datasets can generate synthetic workloads — chat, agentic, and other — for benchmarking LLM inference performance?
A practical map of the tools that generate workloads for benchmarking LLM inference: engine-native load generators (vLLM, SGLang, llama.cpp), trace-driven generators (BurstGPT, Azure traces, ShareGPT/LMSYS replay), statistical generators calibrated to production traffic (ServeGen, FineServe), simulators (LLMServingSim, Frontier, SimLLM), and the emerging agentic-workload tooling (XPerf, SGLang agentic-trace, GuideLLM tool calling). The evidence says workload realism matters — Poisson arrivals and fixed token lengths misrepresent production traffic — and that agentic workloads are the least standardized but fastest-moving category. Confidence is moderate: repo capabilities were verified from primary docs fetched this session, but several tools change weekly and one major tool (NVIDIA GenAI-Perf) was unretrievable.
Selling infrastructure to inference providers: what a neocloud can offer without competing
Which inference providers should an AI-factory builder target, at which infrastructure layer, and what must it know about inference to sell them capacity without becoming their competitor?
The inference-provider market splits into open-weight GPU hosts (Together, Fireworks, DeepInfra, Baseten, Modal — they rent nearly all their capacity and compete on serving software) and closed-weight labs (OpenAI, Anthropic, Google, Meta, xAI — they rent at enormous scale via take-or-pay contracts but vertically integrate software and increasingly silicon). The evidence says a neocloud should target the open-weight hosts at the fleet/facility and control-plane layers — power, cooling, density, grid, orchestration that respects the customer's serving engine — and never compete at the serving-engine layer, which is the customer's moat. Firmus's public record (AI FactoryOS, HyperCube, Model-to-Grid, a Fireworks partnership) already points this way. Confidence is moderate: the market facts are industry-reported, and several claimed product names could not be verified in any public source.
The LLM compute market — training, fine-tuning, and inference
How is LLM compute demand, revenue, and profit split between training, fine-tuning, and inference, and what does the shift toward inference mean for AI factory builders?
Companies spent $37B on enterprise generative AI in 2025 and Gartner counts $2.59T of total AI spending for 2026, and the mix is shifting decisively from training to inference — inference overtakes training in AI-optimized cloud spend by 2026 and across the wider market by 2029. Fine-tuning is a real but small slice of that spend, agentic workloads are the fastest-growing inference category, and frontier training is multimodal-first even though public accounting of compute-by-modality barely exists. The evidence says an AI factory builder should not sell inference tokens against its own customers, but should become inference-workload-aware as infrastructure — disaggregated prefill and decode, SLO-aware scheduling, and tokens-per-watt are all sellable attributes.
The LLM inference engine playbook: tricks in vLLM and SGLang
Which optimization techniques do the vLLM and SGLang serving engines implement, and in what order should an engineer study them?
LLM serving engines are built from a stack of optimization tricks, each attacking one bottleneck: batching keeps the GPU busy, KV-cache paging and reuse stop memory from fragmenting or being recomputed, faster attention kernels and lower-precision arithmetic cut bytes moved, speculative decoding trades spare compute for fewer serial steps, and prefill-decode disaggregation lets two workloads with opposite resource profiles run on machines tuned for each. This review catalogs the trick list of the two dominant open-source engines, vLLM and SGLang, maps each trick to the paper that introduced it and the bottleneck it addresses, and closes with an ordered self-study path. Confidence is moderate: the mechanisms are well established in the literature, but the engines' own performance claims are vendor-reported and rarely independently benchmarked.
GPU kernel development for LLM inference: Triton, TileLang, FlashMLA, and FlashInfer compared
How do Triton, TileLang, FlashMLA, and FlashInfer compare as GPU kernel development tools for LLM inference, and which use cases does each serve best?
The four tools fall into two families: kernel DSLs and compilers (Triton, TileLang) that generate kernels from Python-like tile programs, and attention kernel libraries and engines (FlashInfer, FlashMLA) that ship pre-built or JIT-compiled CUDA kernels for serving. The evidence shows DSLs are within reach of hand-tuned CUDA on GEMM and attention once tuned (59-105% of vendor baselines depending on workload and effort), while the libraries deliver 29-69% lower inter-token latency than compiler backends in serving benchmarks and dominate production deployment. FlashMLA is uniquely tied to one workload, DeepSeek's MLA, and has no peer-reviewed paper, so it is documented here through the DeepSeek technical reports and third-party kernel evaluations. On the training side the same DSLs power the ecosystem layer (torch.compile, Liger, MOSS's Triton FP8 GEMMs), but the training-kernel frontier — block-sparse MoE, single-kernel distributed MoE, FP8 training — is written in expert CUDA and CUTLASS, and FlashInfer and FlashMLA remain forward-only serving libraries. Confidence is moderate: almost all performance claims come from vendor-authored or single-group benchmarks with no independent head-to-head of all four tools.
Kernel fusion for efficient LLM training and inference
Which kernel-fusion methods improve the efficiency of large language model training and inference, and what gains do they deliver?
Kernel fusion — computing several operations in one GPU kernel to cut memory traffic and launch overhead — is the dominant mechanism behind the efficiency gains of the past six years. The evidence is consistent that attention kernels (FlashAttention and its descendants) deliver 2–4x speedups and near-constant memory in sequence length, that fused training-op kernels (Liger and peers) recover roughly 20% throughput and 60% memory on the training side, and that FP8 pipelines now train 10–75% faster with matched quality. Most evidence is preprint-grade and hardware-specific; the field lacks a shared benchmark, and the newest frontier is using LLMs themselves to write the kernels.
Optimizing large language model inference
Which techniques most improve the throughput, latency, and cost of large language model inference?
Large language model inference is memory-bound, not compute-bound, and every major optimization family attacks the memory wall from a different angle: batching and scheduling to keep GPUs busy, KV-cache management to shrink the state that grows with context, quantization to cut bytes per weight, speculative decoding to trade spare compute for fewer serial steps, and prefill-decode separation to stop two workloads with opposite resource profiles from interfering. The evidence is consistent that these techniques deliver large gains — commonly 2–6x in throughput or latency, occasionally 10x or more in narrow regimes — but nearly every headline number is an "up to" figure measured on the authors' own hardware and workload, and independent benchmarks are only beginning to appear.
Interconnects for large-scale LLM training and inference: RDMA, InfiniBand, NVLink, and the network stack
What does the evidence from 2020 to 2026 show about how interconnect technologies (RDMA, InfiniBand, NVLink, Ethernet/RoCE, and the software stacks built on them) shape the performance, scaling, and cost of large-scale LLM training and inference?
Large-scale LLM training and inference now depend on a hierarchy of interconnects: NVLink and NVSwitch inside servers, InfiniBand or RoCE Ethernet between them, and collective-communication libraries that turn raw fabric into training progress. The 2020-2026 evidence shows the lossless RDMA fabric that carried early GPT-era training is under strain, with congestion-control pathologies and head-of-line blocking documented in both directions, while production operators engineer around it and a lossy, packet-spraying Ultra Ethernet path is emerging. Meanwhile Mixture-of-Experts models and long-context inference have made all-to-all traffic and KV-cache movement the new communication bottlenecks, and NVLink scale-up domains, UALink, and CXL memory pooling are reshaping where computation and memory live. Confidence is moderate: the transport and collective stories are replicated across many independent studies, but the newest hardware (NVLink 4/5, UALink, GB200-class superpods) is covered mainly by vendor-adjacent or preprint evidence.